Mailed questionnaires: length matters.

Professor Jenkinson and colleagues are to be congratulated for their recent randomized controlled trial of the effect on response of embedding the Picker Patient Experience questionnaire (PPE-15) within either a four or a 12 page questionnaire. Mailed questionnaires are widely used for data collection in health research, but because non-response reduces the effective sample size and can introduce bias, the assessment of methods that can increase response are important. Several strategies have been identified that can increase response to mailed questionnaires and a meta-analysis of randomized controlled trials of questionnaire length on response suggests that response can be increased using shorter questionnaires. In this meta-analysis, data are combined from 12 randomized controlled trials of questionnaires with two or more pages compared with longer alternatives. The combined odds ratio for each one page increase was estimated to be 0.98 (95 per cent CI 0.96–0.99). When this result is applied in Professor Jenkinson’s study, it predicts that the odds of response to the 12 page questionnaire will be 0.85 times the odds of response to the four page questionnaire (i.e. 0.98). Since 488 of the 721 four page questionnaires were returned we would therefore expect 463 of the 724 12 page questionnaires to be returned. Professor Jenkinson and colleagues found 461 of the 12 page questionnaires were returned. These data are therefore compatible with the pooled result from the meta-analysis The authors are incorrect in saying that no difference in response rate was found between the two questionnaires used in their study and are wrong to conclude that the length of questionnaire in which the PPE-15 was embedded had no impact on response rate. Their study was powered to detect a 10 per cent absolute reduction in response rate and therefore missed the more plausible reduction of 4 per cent. To reliably detect an effect of this size would require a study with sample sizes of around 2000 in each arm. Whether an increase in response from 64 to 68 per cent is important remains a matter for judgement: some researchers would still not consider studies with response as high as 80 per cent to provide sufficient assurance against bias. Non-response should be reduced or eliminated if study results are to be credible. There is a trade-off between questionnaire length and response rates: the inclusion of more questions to collect more information from participants comes at the cost of non-response, loss of precision and possible bias. Researchers must be sure to only include in their questionnaires that which is necessary to achieve their study objectives. Where there is a choice of questionnaire for a study, the shortest should be used; if a new questionnaire is to be designed, it should be made as short as possible.